UNCLASSIFIED 


Analysis  of  Cost  vs.  Reliability 
Growth  Using  a  Simulation  Test  Bed 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 

Martin  Wayne 
10  June  2008 


Approved  for  Public  Release;  Distribution  is  Unlimited. 


UNCLASSIFIED 


Report  Documentation  Page 

Form  Approved 

0MB  No.  0704-0188 

Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 

VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  0MB  control  number. 

1.  REPORT  DATE 

01 JUN  2008 

2.  REPORT  TYPE 

N/A 

3.  DATES  COVERED 

4.  TITLE  AND  SUBTITLE 

Analysis  of  Cost  vs.  Reliability  Growth  Using  a  Simulation  Test  Bed 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

RDECOM 

8.  PEREORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

10.  SPONSOR/MONITOR’S  ACRONYM(S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release,  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

See  also  ADM202527.  Military  Operations  Research  Society  Symposium  (76th)  Held  in  New  London, 
Connecticut  on  June  10-12,  2008,  The  original  document  contains  color  images. 

14.  ABSTRACT 

15.  SUBJECT  TERMS 

16.  SECURITY  CLASSIEICATION  OE: 

17.  LIMITATION  OE 
ABSTRACT 

uu 

18.  NUMBER 
OE  PAGES 

21 

19a.  NAME  OE 
RESPONSIBLE  PERSON 

a.  REPORT 

unclassified 

b.  ABSTRACT 

unclassified 

c.  THIS  PAGE 

unclassified 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


UNCLASSIFIED 


9/17/2008 


Background 

Simulating  a  developmental  test 
Simulation  details 

Cost  vs.  reliability  growth  examples 
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Background 
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•  Simulation  test  bed  initially  developed  for  examining  various 
reliability  growth  models  and  the  robustness  of  assumptions 

•  Other  potential  applications 

-  Reliability  growth  planning 

-  Examining  cost  and  reliability  growth 

•  Cost  application  used  to  assist  LMI  (Dr.  David  Lee)  in  broader  study 
of  cost  and  reliability  growth 
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Desired  Features  of  Developmental  Test 

Simulation 
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•  Test  event  based  on  calendar  time 

•  Corrective  Action  Periods  (CAPs)  built  into  overall  test 

-  Assumes  no  test  time  within  CAP 

•  Allows  for  initial  failure  rates  to  be  chosen  from  various  parent 
populations 

•  Allows  for  different  corrective  actions  strategies 

-  Various  mode  classifications 

-  Corrective  action  delay  times 

•  Captures  data  typical  of  developmental  testing 

-  Failure  rates 

-  Failure  mode  occurrence  times 

-  Repeat  failure  times  for  each  mode 
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Steps  in  Simulating  a  Developmental  Test 
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1 .  Determine  test  structure  (CAP  vs.  test  intervals) 

2.  Choose  initial  failure  rates  for  each  mode  from  parent  population 

3.  Determine  corrective  actions  strategies 

4.  Choose  Fix  Effectiveness  Factors  (FEFs)  from  parent  distribution 

5.  Simulate  failure  times  based  on  appropriate  failure  rates 
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Test  Calendar  Setup 
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•  User  input 

-  Overall  test  length 

-  CAP  duration 

-  Test  interval  duration 

•  Intervals  are  time  unit  independent 

-  Could  be  in  terms  of  weeks,  months,  etc. 

•  Overall  test  divided  into  Test-CAP-Test-CAP-etc.  subintervals  based 
on  inputs 
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Calendar  Example 
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Overall  test  length  =  22  mths 


Test  CAP  Test  CAP  Test  CAP  Test 
period  (2  mths) 

(4  mths) 


***  Partial  interval  will  be  used  when  overall  test  time  does 

not  divide  evenly 
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•  Failure  modes  divided  into  type  A  and  type  B 

-  A  Modes:  not  addressed  through  corrective  action 

-  B  Modes:  addressed  though  corrective  action 

•  User  chooses  number  of  modes  for  each  type 

•  B  modes  further  divided  into  BC  and  BD  types 

-  BC  modes:  addressed  during  test 

-  BD  modes:  corrective  actions  delayed  to  end  of  overall  test 

•  Subdivision  based  on  corrective  action  strategy 

•  Weighted  coin  determines  percentage  of  BC  vs.  BD 
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Failure  Rates 
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•  Failures  rates  determined  through  sampling  from  parent  population 

•  Gamma,  LogNormal,  Weibull,  Geometric  sequence 

•  Equivalent  mean  and  coefficient  of  variation  used  for  each 
distribution 
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Corrective  Action  Strategy 
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•  Corrective  actions  modeled  as  random  process 

•  Fixes  implemented  in  3  different  ways 

-  Within  CAP  (BC  Mode) 

-  Within  test  period  (BC  Mode) 

-  Delayed  until  end  of  overall  test  (BD  Mode) 

•  Weighted  coin  determines  where  BC  Mode  corrective  actions  occur 

•  Empirical  delay  time  distribution  also  used  to  model  delays  in 
implementing  corrective  actions 

-  Distribution  is  user  defined 

-  More  accurately  models  reality 
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Corrective  Action  Example 
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Failure  #1  (CAP  fix) 

Delay  Time 


Fix  occurs  in  CAP 
after  delay  time 
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Test  CAP  Test  CAP  Test  CAP  Test 
period 
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•  Fix  Effectiveness  Factors  (FEFs)  sampled  from  Beta  distribution 

•  Mean  and  coefficient  of  variation  for  distribution  are  user  input 

•  FEF  applied  to  failure  rate  for  B  modes  after  corrective  action  has 
been  implemented 
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Fix  Effectiveness  Factor  Example 
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Fix  occurs  in  1®’ 
CAP  after  delay 
time 

Rate  =  (1-FEFi)Ui 


Delay  Time 
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Test  CAP  Test  CAP  Test  CAP  Test 
period 


Failure  #1 
Rate  = 


***  Failures  occur  at  reduced  rate  after  fix  is  implemented 
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Potential  Output 
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•  Simulated  data  allows  for  calculation  and  plotting  of  various 
quantities  of  interest 

-  Expected  failure  intensity  due  to  unseen  modes,  Failure  mode  profile, 
Expected  number  of  failure  modes,  etc. 
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•  Overall  test  time  =  30  months 

•  Test  interval  =  4  months 

•  CAP  =  2  months 

•  500  test  hours  per  month  (10000 
hours  total) 

•  Management  Strategy  «  0.90 

•  500  B  Modes,  5  A  Modes 

•  Avg.  FEF  =  0.8 

•  %  BC  Modes  =  0.90 

•  %  CAP  fixes  =  0.95 

•  Delay  time  distribution  uniform  from 
0-4  months 


0  2000  4000  6000  3000 


Test  Time 
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Simulating  Cost 
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•  Additional  inputs  necessary 

-  Unit  test  time  cost  (cost  per  month,  week,  etc.) 

-  Cost  per  corrective  action 

•  Total  expected  cost  (C)  modeled  as  combination  of  test  cost  and 
corrective  action  cost^ 


C(t)  =  (c,xt)+(ccAxM) 

t  =  test  time 

=  unit  test  time  cost 
CcA  =  cost  per  corrective  action 
M  =  expected  number  of  failure  modes 
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Example  1  -  Upper  Bound  on  Reliability 

Growth 
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•  Overall  test  time  =  30  months 

•  Test  interval  =  5  months 

•  CAP  =  1  month 

•  1000  test  hours  per  month  (35000  hours  total) 

•  Management  Strategy  =  0.97 

•  500  B  Modes,  5  A  Modes 

•  Avg.  FEF  =  0.7 

•  %  BC  Modes  =  0.95 

•  %  CAP  fixes  =  0.95 

•  Delay  time  distribution  uniform  from  0-4  months 

•  Initial  MTBF  =  10 

•  Test  time  cost  per  month  =  1 000 

•  Cost  per  corrective  action  =  5000 
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Results  -  Example  1 
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•  Relative  change  in  MTBF  over  time  defined  as  ~  where  M(0)  «  10 

•  Upper  bound  on  reliability  indicates  further  investment  in  testing  may  not  be 
beneficial 


Reallocation  of  testing  funds  to  achieve  higher  initial  MTBF  may  be 

necessary 
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Example  2  -  Increasing  FEF 
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•  Overall  test  time  =  30  months 

•  Test  interval  =  4  months 

•  CAP  =  2  month 

•  500  test  hours  per  month  (10000  hours  total) 

•  Management  Strategy  «  0.90 

•  500  B  Modes,  5  A  Modes 

•  Avg.  FEF  =  0.6,  0.8 

•  %  BC  Modes  =  0.90 

•  %  CAP  fixes  =  0.95 

•  Delay  time  distribution  uniform  from  0-4  months 

•  Initial  MTBF«21 

•  Test  time  cost  per  month  =  500 

•  Cost  per  corrective  action  =  10000 
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Rjektivie  Cliange  in  MTEF 


Results  -  Example  2 
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FEF  =  0.80 


Reallocating  funds  to  increase  FEF  may  result  in  higher  overall  MTBF 


9/17/2008 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 

UNCLASSIFIED  c 


UNCLASSIFIED 


•  Simulation  test  bed  useful  for  examining  reliability  growth  in  various 
applications 

-  Model  suitability 

-  Growth  planning 

-  Cost  vs.  reliability  growth 

•  Upper  bounds  on  reliability  suggest  further  investment  in  testing  may 
not  be  sufficient  to  meet  goals  in  some  cases 

•  Reallocating  funds  to  other  growth  applications  (i.e.  increasing  FEF) 
may  be  more  beneficial  than  increased  testing  in  some  cases 
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